Large-scale computing systems, such as those associated with network-based production services, have become widely available in recent years. Examples of such systems include online merchants, internet service providers, online businesses such as photo processing services, corporate networks, cloud computing services, web-based hosting services, etc. These entities may maintain large numbers of computing devices (e.g., thousands of hosts) which are hosted in geographically separate locations and which are configured to process large quantities (e.g., millions) of client requests daily or even hourly. Complex systems may include many services that interact with one another in varied ways.
Automated testing of such services is an increasingly important part of the software development process. As part of the build process and also as part of the deployment process, a suite of tests may be run to verify the expected operation of the software. However, running a large suite of tests may take a significant amount of time, e.g., tens of minutes to multiple hours. While the software is being tested, a developer may be forced to stop work on the project for the duration of the entire suite of tests. The time taken to run the suite of tests may also limit the number of feedback cycles for a developer in a given period of time. If a developer continues writing code while a program is being tested, only to find that the tests eventually fail, then the developer may need to spend additional time to undo or modify the recent work to address the issues revealed by the failed tests. Accordingly, in large organizations with thousands of developers, automated testing of software builds may result in developer inefficiency on a large scale.
While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning “having the potential to”), rather than the mandatory sense (i.e., meaning “must”). Similarly, the words “include,” “including,” and “includes” mean “including, but not limited to.”